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SYSTEM AI^^ETHOD FOR SAVING HAND\^^TING AS AN 

ANNOTATION IN A SCANNED DOCUMENT 

TECHNICAL FIELD 

5 The present invention relates generally to electronic scanning devices, 

and, more particularly, to a system and method for saving handwriting in a 
scanned document as an annotation in the document. 

BACKGROUND OF THE INVENTION 

10 Scanning devices are useful in many applications where it is desirable to 

transfer an image from printed form into electronic form. Scanners capable of 
reading and converting a page into electronic format have been available for quite 
some time. Typically, a scanner will electronically read a page and electronically 
store the information for later presentation and use. 

15 Typically, a scanner will electronically read aL page, classify the different 

types of images on the page and electronically store the information for later 
presentation and use. The types of classifications of a scanned page typically 
include text, photographs, drawings, charts, tables, business graphics, equations, 
handwriting, logos, etc. Alternatively, a user can classify these different parts of a 

20 scanned image into regions after the scanner has scanned the page. Some 

scanners are capable of determining the classifications of particular regions of a 
scanned page in accordance with predetermined instructions. 

Oftentimes, a scanned image includes handwritten comments written 
directly on the page that is scanned. Conventional scanning techniques and 

25 systems scan the entire page and present to a user all of the information in the 
scanned image that the scanning system deems as relevant. If there is 
handwriting on the original image, the scanning system might attempt to use an 
optical character recognition (OCR) program on the handwriting, but this usually 
results in the handwriting being interpreted as stray markings (i.e., garbage). 

30 Alternatively, the handwriting may be saved as an image and presented to the 
user. In yet another alternative, the handwriting detected in a scanned image 
may be classified as unusable and eliminated from the scanned image by the 
processing system. 
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Unfortunatel^fcs leads to scanned images that ar^^^ttered and leave the 
user no manner in which to conveniently remove the handwriting from the 
scanned image. In a worst case, important handwriting may be eliminated 
altogether from the scanned image by the processing system. 
5 Therefore, it would be desirable to have the ability to control the manner in 

which handwriting on a scanned image is saved, and have the ability to present 
the user with options regarding the manner in which the handwriting is presented 
in the scanned image. 

10 SUMMARY OF THE INVENTION 

The invention provides a system and method for saving handwriting in a 
scanned image as an annotation in the document. 

Architecturally, the present invention can be conceptualized as a system 
J for saving handwriting in a scanned image as an annotation. The system 

m 15 includes a scanner, a document analysis and processing software component in 
!il communication with the scanner, and automatic annotation logic in 

£3 communication with the document analysis and processing software component. 

■ 3 The automatic annotation logic is configured to detect a region in a scanned 

% image, the region including handwriting, and save the handwriting as an 

=3 20 annotation in a document generated from the scanned image. 
;3 The present invention may also be conceptualized as a method for saving 

handwriting in a scanned image as an annotation. The method includes the 
steps of scanning an image in a scanner, detecting a region in the scanned 
image, the region including handwriting, and saving the handwriting in the 
25 detected region as an annotation in a document generated from the scanned 
image. 

The invention has numerous advantages, a few of which are delineated, 
hereafter, as merely examples. 

An advantage of the invention is that it enables a user of a scanner to 
30 control the manner in which handwriting or notations are saved and presented in 
the document that is generated from the scanned image. 
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iffl|e of the invention is that it enables^ie 



Another advaiS^e of the invention is that it enables^e user of a scanner 
to control the manner in which handwriting or notations are presented in a 
preview image shown to the user. 

Another advantage of the invention is that it improves the appearance of a 
5 document saved from a scanned image that includes handwriting or notations. 

Another advantage of the invention is that it allows the user of a scanner 
to view a document generated from a scanned image that contained handwriting 
or notations without the handwriting or notations so that the look of the original 
document may be preserved and viewed. 
10 Another advantage of the invention is that because the handwriting is 

saved as an annotation it is always available to the user in electronic format. 

Another advantage of the invention is that it is simple in design and easily 
implemented on a mass scale for commercial production. 
J Other features and advantages of the invention will become apparent to one 

'i 15 with skill in the art upon examination of the following drawings and detailed 
=~ description. These additional features and advantages are intended to be included 

:? herein within the scope of the present invention. 

- -z~ 

! BRIEF DESCRIPTION OF THE DRAWINGS 

s ~S 

; 3 20 The present invention, as defined in the claims, can be better understood 

I! with reference to the following drawings. The components within the drawings 

are not necessarily to scale relative to each other, emphasis instead being placed 
upon clearly illustrating the principles of the present invention. 

Fig. 1 is a schematic view of an exemplar scanner and computer system in 
25 which the automatic annotation logic of the present invention resides; 

Fig. 2 is a schematic view illustrating the user interface and analysis and 
processing logic of Fig. 1 in which the logic of the present invention resides; 

Fig. 3 is a flow diagram illustrating the operation of the automatic 
annotation logic of Figs. 1 and 2; and 
30 Fig. 4 is a graphical representation of one embodiment of the automatic 

annotation logic of Figs. 1 and 2. 
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DETAILED & 




IPTION OF THE PREFERRE 




ODIMENT 



The automatic annotation logic of the present invention can be 
implemented in software, hardware, or a combination thereof. In a preferred 
embodiment(s), the automatic annotation logic is implemented in software or 
5 firmware that is stored in a memory and that is executed by a suitable 

instruction execution system (microprocessor). If implemented in hardware, as 
in an alternative embodiment, the automatic annotation logic can implemented 
with any or a combination of the following technologies, which are all well 
known in the art: a discrete logic circuit(s) having logic gates for implementing 

10 logic functions upon data signals, an application specific integrated circuit 
having appropriate logic gates, a programmable gate array(s) (PGA), a field 
programmable gate array (FPGA), etc. 

Furthermore, the automatic annotation software, which comprise an 
ordered listing of executable instructions for implementing logical functions, can 

15 be embodied in any computer-readable medium for use by or in connection with 
an instruction execution system, apparatus, or device, such as a computer-based 
system, processor-containing system, or other system that can fetch the 
instructions from the instruction execution system, apparatus, or device and 
execute the instructions. 

20 In the context of this document, a "computer- readable medium" can be any 

means that can contain, store, communicate, propagate, or transport the 
program for use by or in connection with the instruction execution system, 
apparatus, or device. The computer readable medium can be, for example but 
not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or 

25 semiconductor system, apparatus, device, or propagation medium. More specific 
examples (a nonexhaustive list) of the computer-readable medium would include 
the following: an electrical connection (electronic) having one or more wires, a 
portable computer diskette (magnetic), a random access memory (RAM) 
(magnetic), a read-only memory (ROM) (magnetic), an erasable programmable 

30 read-only memory (EPROM or Flash memory) (magnetic), an optical fiber 
(optical), and a portable compact disc read-only memory (CDROM) (optical). 
Note that the computer-readable medium could even be paper or another 
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)^^hich the program is printed, as tl^^< 



suitable medium up^^hich the program is printed, as th^program can be 
electronically captured, via for instance optical scanning of the paper or other 
medium, then compiled, interpreted or otherwise processed in a suitable manner 
if necessary, and then stored in a computer memory. 

5 Moreover, while the automatic annotation logic will be illustrated 

hereafter with respect to the recognition of handwriting in a scanned document, 
the automatic annotation logic is useful for automatically recognizing other 
attributes of a scanned image, and saving them as annotations to the document, 
for example but not limited to drawings, photographs, equations, graphics, 

10 lineart, etc. 

Turning now to Fig. 1, shown is a schematic view of an exemplar scanner 
and computer system 100 in which the automatic annotation logic 110 of the 
present invention resides. 

Illustratively, scanner 11 scans a document placed therein in cooperation 

15 with computer 12. Computer 12 can be any general purpose computer that is 
capable of connecting to and executing the software that enables scanner 11 to 
function. Illustratively, computer 12 is a personal computer, however, computer 
12 may be any computer capable of communicating with scanner 11. A scanned 
image is captured by data capture element 14 located within computer 12. The 

20 scanned image data is illustratively stored in random access memory (RAM) 16. 
RAM 16 communicates with analysis and processing logic 17, user interface 13 
and microprocessor (uP) 25 over bus 24. 

Analysis and processing logic 17, together with automatic annotation logic 
110, is illustratively the logic that operates in conjunction with scanner 11 to 

25 determine the region of the scanned image within which to inspect for 

handwriting in the scanned image. Alternatively, analysis and processing logic 
17, together with automatic annotation logic 110, may inspect the image for the 
presence of a notation, such as a "POST-IT®" note on the document. If such a 
notation is detected on the scanned document, the logic will analyze any 

30 handwriting present on the notation and automatically save that handwriting as 
an annotation in the document saved from the scanned document, as will be 
described more fully below. 
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All identifyin^Wements of a scanned image, such a: 





e, location and 



statistics of the scanned image are captured by data capture element 14 and 
stored as captured data in RAM 16. Furthermore, analysis and processing logic 
17 in conjunction with uP 25 is the underlying processing engine that maintains 
5 the scanned image data. Analysis and processing logic 17 includes automatic 
annotation logic 110, which will be described in greater detail below with respect 
to Fig. 2. Automatic annotation logic 110 resides within analysis and processing 
logic 17, which communicates with data manager 21 over bus 24 and connection 
28. Data manager 21 communicates with bus 24 over connection 28 in order to 

10 access the data stored in RAM 16 in order to perform the preview scan operation 
or other post analysis tasks. Preview scan is a feature of many scanner 
programs and enables a user to preview the scanned image on a computer 
monitor prior to scanning and saving the document. In this manner, the user of 
a scanner may decide the appearance of the document prior to saving. As will be 

15 discussed below, some of the document processing features of the invention will 
interact with the preview scan feature. However, the present invention is not 
reliant upon the preview scan operation. Indeed, the preview scan operation 
may be omitted in some applications, for example when batch processing. Post 
analysis tasks may include, for example, printing, faxing, optical character 

20 recognition, etc. 

In one particular embodiment of the invention, the automatic annotation 
logic 110 enables the analysis and processing logic 17 to recognize a 
predetermined portion, or region, of a scanned image in which a user has 
indicated the presence of text. For example, a user might instruct the scanner 

25^ software to inspect the upper right hand portion of a scanned image for 

handwriting, which will be saved as an annotation to the scanned document. 

Typically, the scanner analysis and processing logic will divide the 
scanned document into different regions, depending upon the information 
contained in the document. For example, assuming that there are both text and 

30 a photograph in the document, the scanner analysis and processing logic 

recognizes the two different types of information (text and a photograph) and 
assigns each of the different types of information to a region. The text is one 
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t^^ph is another region. Alternative^^ 



region and the phot^^iph is another region. Alternative!^^ user of the scanner 
may manipulate and assign different regions to the different types of 
information. The technology used to define and assign different regions will not 
be described in detail. Many other types of information may be included in a 
5 document and assigned a region and an information type. 

In another embodiment of the invention, automatic annotation logic 110 
enables the analysis and processing logic 17 to recognize the presence of 
handwriting in the scanned document, whether alone or on a notation on the 
document, and save the handwriting as an annotation to the document. If a 
10 notation is detected, then the automatic annotation logic 110 will analyze the 
notation for handwriting and save that information as an annotation. 

Illustratively, by indicating a particular portion of the scanned document 
to the analysis and processing logic 17, a user may indicate a region of interest 
=3 by interacting with the software and using a mouse and pointer to draw a 

=i 15 bounding box around a portion of the scanned image that the user desires the 
It software to analyze. Alternatively, a user could draw a bounding box and use 

the above-mentioned region selection method to type the region of interest as 
r "handwriting". When scanned, the software analyzes the previously indicated 

7j region of interest for handwritten text and saves that handwriting as an 

;f! 20 annotation to the document. 

'"-4 

;3 Similarly, the analysis and processing logic 17, together with the 

automatic annotation logic 110, may use a color and hue recognition technique to 
locate a notation on the page. Any writing on the notation would then be saved 
as an annotation to the scanned image. 

25 Also included in computer 12 is user interface 13, which illustratively 

includes preview scan block 18. Preview scan 18 allows a user of a scanner to 
view the document to be scanned prior to final scanning, or otherwise, prior to 
sending the appropriate scanned data regions to downstream destinations 
(applications, storage, etc.). Preview scan 18 outputs the scanned image on 

30 connection 32 for output to a user on display 19. 

Fig. 2 is a schematic view illustrating the user interface 13 and analysis 
and processing logic 17 of Fig. 1, in which the automatic annotation logic 110 of 
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the present inventior^fcides. User interface 13 includes p^few scan block 18 
and user control 34. User control 34 may include any available means or method 
for interacting with the computer, and commonly includes a mouse, a stylus, or a 
computer keyboard. User control 34 can be used to communicate the region of the 
scanned page, in which the user has inserted handwriting. As discussed above, in 
some applications such as batch processing, the preview scan operation, and any 
user interaction may be omitted. 

Automatic annotation logic 110 resides within analysis and processing logic 
17 and contains the logic necessary to allow a scanner to automatically recognize 
handwriting in a scanned image. Handwriting can be recognized and interpreted 
whether written directly on the document, or whether written on a notation 
applied to the document. Automatic annotation logic 110 includes analysis code 
120, region manager 125 and document processing code 130. 

If a region in a scanned image is defined by a user as the region in which 
handwriting is located, the annotation detection logic 160 uses handwriting 
detection logic 145 to detect handwriting in the selected region. Once the 
handwriting is detected, handwriting recognition logic 140 is used to analyze the 
handwriting so that the handwriting may be saved as an annotation to the 
scanned image by the annotation processing logic 170. 

Alternatively, the automatic annotation logic 110 can be configured to 
search the entire document for the presence of handwriting. In such a case, the 
annotation detection logic 160 uses handwriting detection logic 145 to detect 
handwriting anywhere in the document. Then, in similar fashion to that described 
above, once the handwriting is detected, handwriting recognition logic 140 is used 
to analyze the handwriting so that the handwriting may be saved as an 
annotation to the scanned image. 

If the scanner software is configured by a user to search the scanned image 
for a notation, then analysis code 120 uses notation detection logic 150 to detect 
the presence of the notation. Notation detection logic 150 includes color 
recognition logic 135, which can detect the presence of color, such as in the case of 
a notation having a color different than that of the scanned page. Once the 
notation is detected, the document processing code 130 uses handwriting 
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) r^ead the handwriting from the notafilSn, 



recognition logic 140 r^ead the handwriting from the notatWn, and uses 
annotation processing logic 170 to save the handwriting as an annotation to the 
scanned document. 

Document processing code 130 also includes handwriting/notation removal 
5 logic 180. Handwriting/notation removal logic 180 may optionally be used to 
remove the appearance of the handwriting or notation in the preview image 
presented to the user via the above described preview scan feature. One possible 
way to do this is to fill the region boundaries with the background color of the 
document (usually white). Although described herein as handwriting/notation 
10 removal logic 180, the logic can be used to clear any region in the document. 

In addition, handwriting/notation removal logic 180 may also optionally 
be used to remove the appearance of the handwriting or notation from the saved 
document. Typically, the use of handwriting/notation removal logic 180 on 
handwriting or notations on the preview image would help the user of the 
15 scanner decide whether or not they would like to apply this removal logic to the 
saved document. 

User interface 13 communicates with analysis and processing logic 17 over 
bus 24. If the user desires, preview scan 18 displays the handwriting region 
detected by the automatic annotation logic 110 on display 19. Once the document 

20 is viewed on display 19 during the preview scan operation 18, the user has an 
opportunity to manipulate the handwriting region. 

Fig. 3 is a flow diagram collectively illustrating the operation of the 
automatic annotation logic of Figs. 1 and 2. 

The flow chart of Fig. 3 shows the architecture, functionality, and 

25 operation of a possible implementation of the automatic annotation software of 
Figs. 1 and 2. In this regard, each block represents a module, segment, or 
portion of code, which comprises one or more executable instructions for 
implementing the specified logical function(s). It should also be noted that in 
some alternative implementations, the functions noted in the blocks may occur 

30 out of the order noted in Fig. 3. For example, two blocks shown in succession in 
Fig. 3 may in fact be executed substantially concurrently or the blocks may 
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sometimes beexecutSKn the reverse order, depending upoKhe functionality 
involved, as will be further clarified hereinbelow. 

With reference to Fig. 3, in block 201, a document is scanned. In block 
202, the analysis and processing logic 17 discovers the different regions on the 

5 scanned document. The regions can be anywhere on the document. For 

illustration purposes, and to be described with respect to Fig. 4, the area of the 
document in which handwriting was found is the upper right hand portion of a 
page. This step can be set either by user input (i.e., a user selecting an area on a 
scanned page through the use of a mouse and pointer), or may be determined by 

10 the analysis and processing logic 17. 

In block 204, it is determined whether the region under analysis is a 
notation. This is accomplished through the use of notation detection logic 150 of 
Fig. 2. As used in this embodiment and as described above, a notation can be a 
note, the type of which is frequently used for leaving removable notes on a page. 

15 A "POST-IT®" note is envisioned in this embodiment because such notes tend to 
be of different color than the page upon which they are applied. However, the 
invention should not be construed as limited to any particular type of notation. 
For example, notes such as these are typically made in noticeable colors of 
yellow, green, pink, or blue. In this manner, it is possible to detect their 

20 presence through the use of color recognition logic 135 such as that described 
above with respect to Fig. 2. Color recognition and the detection thereof is 
known to those having ordinary skill in the art and will not be explained in 
detail herein. 

Other techniques for determining the presence of a notation are also 
25 possible. For example, logic to detect the presence of a note having a particular 
shape on a scanned page may be employed by notation detection logic 150 to 
detect the presence of a notation. 

If it is determined that the region of interest is a notation, then, in block 
208, handwriting recognition logic 140 (Fig. 2) is used to process the handwriting 
30 on the notation. Once the handwriting is processed, then in block 209 the 

automatic annotation logic 110 saves the handwriting as an annotation to the 
scanned document. If, in block 204, it is determined that the region under 
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t^^n, then in block 206 it is determine^vl 



analysis is not a not^ron, then in block 206 it is determiri^^vhether the region 
of interest includes any handwriting. This is accomplished by using handwriting 
detection logic 145 (Fig. 2) contained within analysis code 120. If handwriting 
detection logic 145 discovers handwriting in the region, then in block 208, 
5 handwriting recognition logic 140 (Fig. 2) is used to process the handwriting. 
Once the handwriting is processed, then in block 209 the automatic annotation 
logic 110 saves the handwriting as an annotation to the scanned document. 

After the handwriting is saved as an annotation in block 209, then, in 
block 210, optional processes can be performed by the analysis and processing 

10 logic 17 (Fig. 1). For example, handwriting/notation removal logic 180 can be 
used to eliminate the appearance of the region and/or the appearance of the 
handwriting from the preview scan image. In block 212, the analysis and 
processing logic 17 (Fig. 1) can invoke the handwriting/notation removal logic 
180 to eliminate the appearance of the region and/or the handwriting from the 

15 document generated and saved from the scanned image. One possible way to do 
this is to fill the region boundaries with the background color of the document 
(usually white). After the performance of the optional processes in blocks 210 
and 212, it is determined, in block 211, whether there are any more regions to 
process. 

20 If it is determined in block 206 that the region of interest is not a 

handwriting region, then in block 207 the region is processed as appropriate for 
the type of information contained therein. For example, if the region contains 
text, an optical character recognition process operates on the text in accordance 
with techniques known in the art. If the region contains a photograph, the 

25 photograph is raster scanned and saved in accordance with techniques known in 
the art. 

In block 211 it is determined whether there are any more regions to 
analyze. If not, then the process ends. If there are additional regions to analyze, 
then the process returns to block 204. 
30 Fig. 4 is a graphical representation of one embodiment of the automatic 

annotation logic 110 of Figs. 1 and 2. Fig. 4 illustrates sample page 250 on 
which text 258 is printed, written, or otherwise applied. Also on page 250 is 
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bounding box 255 




ding box 255 can be indicated by ^fser of the scanner 




by using a mouse and pointer to indicate the desired area, or may be determined 
by the analysis and processing logic 17. Bounding box 255 encloses a region 257, 
within which the automatic annotation logic 110 and the handwriting 
5 recognition logic 140 will analyze to determine whether any handwriting is 
present. 

Regardless of the manner in which handwriting 258 is detected, once 
detected it is automatically saved as an annotation to the scanned document by 
automatic annotation logic 110 (Fig. 2). 
10 In the example shown in Fig. 4, writing 258 is present within region 257. 

Writing 258 includes the text "great job". By searching region 257 and 
recognizing the text 258, automatic annotation logic 110 will save the text "great 
job" as an annotation to the scanned document. In addition, the region that 
| contained the handwriting may be removed from the final document, so as to 
2 15 reduce clutter in the document. In this manner, when the document that is 
* generated from the scanned page is displayed to a user, the user may choose to 

I view the comment "great job" or hide the comment from view. 

When the handwriting is saved as an annotation, the user may easily 
f! manipulate the handwriting using optional processing applications. For 

ill 

1 20 example, the handwriting saved as an annotation may not be visible in the 
1 scanned document until a user desires to read the annotation. In such a case, 
~ the presence of an annotation at a particular place in a document is indicated to 
the user by, for example, a superscript identifier or a highlighted color over a 
portion of the document. If the user desires to read the annotation, the user may 
25 indicate this by placing the cursor over the indicated portion of the document, 
thereby signaling to the application that the user wishes to view the annotation. 
This is merely one manner in which the annotation may be maintained as 
invisible in the document. Many other techniques are contemplated. 



30 Figs. 1 and 2, page 250 also includes notation 265. As illustrated in Fig. 4, 

notation 265 includes the handwriting "great job". When page 250 is scanned, 
notation recognition logic 150 (Fig. 2) will detect notation 265. Notation 265 will 



In an alternative embodiment of the automatic annotation logic 110 of 
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then be analyzed by TRPcomatic annotation logic 110 to automatically save the 
handwriting on the notation as an annotation to the scanned document. Once 
notation recognition logic 150 detects notation 265, handwriting recognition logic 
140 (Fig. 2) will analyze the writing and save the handwriting as an annotation. 
Optionally, as mentioned above, the automatic annotation logic 110 may operate 
on the notation so that the appearance of the notation and handwriting is 
removed from the document. The information that the notation and handwriting 
conveyed is saved as an annotation. 

It will be obvious to those skilled in the art that many modifications and 
variations may be made to the preferred embodiments of the present invention, 
as set forth above, without departing substantially from the principles of the 
present invention. For example, the system and method for saving handwriting 
as an annotation in a scanned document can be implemented using various 
scanning and computing products and can be applied to other types of regions. 
All such modifications and variations are intended to be included herein within 
the scope of the present invention, as defined in the claims that follow. 
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